{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "run_review_scorer.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9aTVeuq-azu",
        "colab_type": "text"
      },
      "source": [
        "# NLP Review Scorer (Toy Version)\n",
        "\n",
        "**Disclaimer: This is only a toy. You should seriously treat your rebuttal despite the what scores are given below. Wish you good luck with your paper submission!**\n",
        "\n",
        "I know some of you are thinking about how to convert paper review to a numerical score.\n",
        "Yes, the time has come.\n",
        "\n",
        "In this notebook, you will be able to convert your review to overall score (hopefully in range 1~5).\n",
        "\n",
        "I assume that you have followed the pre-steps on GitHub: https://github.com/ymcui/NLP-Review-Scorer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23lw9AXX--mT",
        "colab_type": "text"
      },
      "source": [
        "### Step 1: Mount your Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FXE6JqBjCA-z",
        "colab_type": "code",
        "outputId": "d5ab497f-7140-490f-ec96-119116f59d88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m6HeCrot_gD2",
        "colab_type": "text"
      },
      "source": [
        "### Step 2: Unzip model to Colab\n",
        "Note that, the model will be updated occasionally according to the prediction performance. I will only keep the latest model here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "18Yf8Bqk_uEX",
        "colab_type": "code",
        "outputId": "3ad6be09-98d5-4e52-ed69-35dbdec8dc86",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "!unzip -n /content/drive/My\\ Drive/review_model_0711.zip -d /content/bert"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  /content/drive/My Drive/review_model_0711.zip\n",
            "   creating: /content/bert/model0711/\n",
            "  inflating: /content/bert/model0711/vocab.txt  \n",
            "  inflating: /content/bert/model0711/model.ckpt-0.meta  \n",
            "  inflating: /content/bert/model0711/bert_config.json  \n",
            "  inflating: /content/bert/model0711/model.ckpt-0.index  \n",
            "  inflating: /content/bert/model0711/model.ckpt-0.data-00000-of-00001  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtNRD4ZVEode",
        "colab_type": "text"
      },
      "source": [
        "## Step 3: Upload dependency files (from GitHub)\n",
        "Clike 'upload' button to upload:\n",
        "- modeling.py\n",
        "- run_classifier.py\n",
        "- optimization.py\n",
        "- tokenization.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8jQx-wIARWx",
        "colab_type": "text"
      },
      "source": [
        "## Step 4: Input your review and RUN!\n",
        "\n",
        "Note that, it is better to remove '\\n' in your review before copy to `review_text` field.\n",
        "\n",
        "Be careful not to remove quote marks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KvvM-JcowNMq",
        "colab_type": "code",
        "outputId": "0f14b609-c9a0-4e88-dea8-df04bd381eea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# -*- coding: utf-8 -*-\n",
        "\"\"\"run_squad_on_colab.ipynb\n",
        "\n",
        "Automatically generated by Colaboratory.\n",
        "\"\"\"\n",
        "\n",
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "'''\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU.\n",
        "'''\n",
        "\n",
        "\n",
        "\"\"\"### Prepare and import BERT modules\n",
        "With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. Alternatively, you can install BERT using pip (!pip install bert-tensorflow).\n",
        "\"\"\"\n",
        "\n",
        "# import python modules defined by BERT\n",
        "import sys\n",
        "import collections\n",
        "import modeling\n",
        "import optimization\n",
        "import tokenization\n",
        "from run_classifier import ReviewProcessor, file_based_convert_examples_to_features, file_based_input_fn_builder, model_fn_builder, PaddingInputExample\n",
        "import numpy as np\n",
        "\n",
        "review_text = \"The paper was rather bad that I don't want to see it again. The idea was trivial and the evaluations are not convincing to me at all. We should reject this paper or I won't review for this venue in the future,\" #@param {type:\"raw\"}\n",
        "review_sample= [\"emnlp2019\",\"0\",\"0\",review_text]\n",
        "\n",
        "vocab_file='/content/bert/model0711/vocab.txt'\n",
        "bert_config_file='/content/bert/model0711/bert_config.json'\n",
        "init_checkpoint='/content/bert/model0711/model.ckpt-0'\n",
        "\n",
        "do_train=False\n",
        "do_predict=True #@param [\"False\", \"True\"] {type:\"raw\"}\n",
        "train_batch_size=32\n",
        "predict_batch_size=8\n",
        "eval_batch_size=8\n",
        "max_seq_length=512\n",
        "save_checkpoints_steps=5000\n",
        "do_lower_case=False\n",
        "use_tpu=False\n",
        "warmup_proportion=0.1\n",
        "learning_rate=1e-5\n",
        "num_train_epochs=1\n",
        "\n",
        "def main():\n",
        "  output_dir = '/content/result'\n",
        "  tf.gfile.MakeDirs(output_dir)\n",
        "  print('***** Model output directory: {} *****'.format(output_dir))\n",
        "\n",
        "  ########################################################################\n",
        "\n",
        "  tf.logging.set_verbosity(tf.logging.INFO)\n",
        "\n",
        "  bert_config = modeling.BertConfig.from_json_file(bert_config_file)\n",
        "  tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
        "\n",
        "  #validate_flags_or_throw(bert_config)\n",
        "  \n",
        "  processor = ReviewProcessor()\n",
        "\n",
        "  tf.gfile.MakeDirs(output_dir)\n",
        "  tpu_cluster_resolver = None\n",
        "  if use_tpu:\n",
        "    tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "  is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
        "  run_config = tf.contrib.tpu.RunConfig(\n",
        "      cluster=tpu_cluster_resolver,\n",
        "      model_dir=output_dir,\n",
        "      save_checkpoints_steps=5000,\n",
        "      keep_checkpoint_max=2,\n",
        "      tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "          iterations_per_loop=1000,\n",
        "          num_shards=8,\n",
        "          per_host_input_for_training=is_per_host))\n",
        "\n",
        "\n",
        "  train_examples = None\n",
        "  num_train_steps = None\n",
        "  num_warmup_steps = None\n",
        "\n",
        "  model_fn = model_fn_builder(\n",
        "      bert_config=bert_config,\n",
        "      init_checkpoint=init_checkpoint,\n",
        "      learning_rate=learning_rate,\n",
        "      num_train_steps=num_train_steps,\n",
        "      num_warmup_steps=num_warmup_steps,\n",
        "      use_tpu=use_tpu,\n",
        "      use_one_hot_embeddings=use_tpu)\n",
        "\n",
        "  # If TPU is not available, this will fall back to normal Estimator on CPU\n",
        "  # or GPU.\n",
        "  estimator = tf.contrib.tpu.TPUEstimator(\n",
        "      use_tpu=use_tpu,\n",
        "      model_fn=model_fn,\n",
        "      config=run_config,\n",
        "      train_batch_size=train_batch_size,\n",
        "      eval_batch_size=eval_batch_size,\n",
        "      predict_batch_size=predict_batch_size)\n",
        "\n",
        "  if do_predict:\n",
        "    predict_examples = processor.get_single_examples([review_sample])\n",
        "    num_actual_predict_examples = len(predict_examples)\n",
        "    if use_tpu:\n",
        "      # TPU requires a fixed batch size for all batches, therefore the number\n",
        "      # of examples must be a multiple of the batch size, or else examples\n",
        "      # will get dropped. So we pad with fake examples which are ignored\n",
        "      # later on.\n",
        "      while len(predict_examples) % predict_batch_size != 0:\n",
        "        predict_examples.append(PaddingInputExample())\n",
        "\n",
        "    predict_file = os.path.join(output_dir, \"predict.tf_record\")\n",
        "    file_based_convert_examples_to_features(predict_examples,\n",
        "                                            max_seq_length, tokenizer,\n",
        "                                            predict_file)\n",
        "\n",
        "    tf.logging.info(\"***** Running prediction*****\")\n",
        "    tf.logging.info(\"  Num examples = %d (%d actual, %d padding)\",\n",
        "                    len(predict_examples), num_actual_predict_examples,\n",
        "                    len(predict_examples) - num_actual_predict_examples)\n",
        "    tf.logging.info(\"  Batch size = %d\", predict_batch_size)\n",
        "\n",
        "    predict_drop_remainder = True if use_tpu else False\n",
        "    predict_input_fn = file_based_input_fn_builder(\n",
        "        input_file=predict_file,\n",
        "        seq_length=max_seq_length,\n",
        "        is_training=False,\n",
        "        drop_remainder=predict_drop_remainder)\n",
        "\n",
        "    result = estimator.predict(input_fn=predict_input_fn)\n",
        "    tf.logging.info(result)\n",
        "\n",
        "    output_predict_file = os.path.join(\"./test_results.tsv\")\n",
        "    with tf.gfile.GFile(output_predict_file, \"w\") as writer:\n",
        "      num_written_lines = 0\n",
        "      tf.logging.info(\"***** Predict results *****\")\n",
        "      writer.write(\"paper\\trecommendation\\tconfidence\\n\")\n",
        "      for (i, prediction) in enumerate(result):\n",
        "        if i >= num_actual_predict_examples:\n",
        "          break            \n",
        "        probabilities = prediction[\"probabilities\"]\n",
        "        output_line = \"\\t\".join(\n",
        "            str(class_probability)\n",
        "            for class_probability in probabilities) + \"\\n\"\n",
        "        output_line = predict_examples[i].guid + \"\\t\" + output_line\n",
        "        writer.write(output_line)\n",
        "        num_written_lines += 1\n",
        "    tf.logging.info(\"***********REVIEW**************\")\n",
        "    tf.logging.info(review_text)\n",
        "    tf.logging.info(\"***********SCORE***************\")\n",
        "    tf.logging.info(\"paper\\trecommendation\\tconfidence\")\n",
        "    tf.logging.info(output_line)\n",
        "    tf.logging.info(\"********************************\")\n",
        "    assert num_written_lines == num_actual_predict_examples\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "  main()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0711 07:24:57.391819 139746709817216 estimator.py:1984] Estimator's model_fn (<function model_fn at 0x7f18fb8b11b8>) includes params argument, but params are not passed to Estimator.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "***** Model output directory: /content/result *****\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0711 07:24:57.394778 139746709817216 estimator.py:209] Using config: {'_save_checkpoints_secs': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 2, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f18fa6d1810>, '_model_dir': '/content/result', '_protocol': None, '_save_checkpoints_steps': 5000, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2), '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_cluster': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': None, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_master': ''}\n",
            "I0711 07:24:57.397031 139746709817216 tpu_context.py:209] _TPUContext: eval_on_tpu True\n",
            "W0711 07:24:57.398132 139746709817216 tpu_context.py:211] eval_on_tpu ignored because use_tpu is False.\n",
            "I0711 07:24:57.401592 139746709817216 run_classifier.py:362] Writing example 0 of 1\n",
            "I0711 07:24:57.408087 139746709817216 run_classifier.py:336] *** Example ***\n",
            "I0711 07:24:57.409317 139746709817216 run_classifier.py:337] guid: emnlp2019\n",
            "I0711 07:24:57.412009 139746709817216 run_classifier.py:339] tokens: [CLS] The paper was rather bad that I don ' t want to see it again . The idea was trivial and the evaluation ##s are not convincing to me at all . We should reject this paper or I won ' t review for this venue in the future , [SEP]\n",
            "I0711 07:24:57.414088 139746709817216 run_classifier.py:340] input_ids: 101 1109 2526 1108 1897 2213 1115 146 1274 112 189 1328 1106 1267 1122 1254 119 1109 1911 1108 23594 1105 1103 10540 1116 1132 1136 13870 1106 1143 1120 1155 119 1284 1431 16589 1142 2526 1137 146 1281 112 189 3189 1111 1142 6590 1107 1103 2174 117 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0711 07:24:57.416136 139746709817216 run_classifier.py:341] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0711 07:24:57.418236 139746709817216 run_classifier.py:342] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0711 07:24:57.420336 139746709817216 run_classifier.py:343] label: 0.0 0.0 \n",
            "I0711 07:24:57.424509 139746709817216 <ipython-input-6-fedbeaf019fe>:138] ***** Running prediction*****\n",
            "I0711 07:24:57.426558 139746709817216 <ipython-input-6-fedbeaf019fe>:141]   Num examples = 1 (1 actual, 0 padding)\n",
            "I0711 07:24:57.427694 139746709817216 <ipython-input-6-fedbeaf019fe>:142]   Batch size = 8\n",
            "I0711 07:24:57.429703 139746709817216 <ipython-input-6-fedbeaf019fe>:152] <generator object predict at 0x7f18ad98ad70>\n",
            "I0711 07:24:57.431761 139746709817216 <ipython-input-6-fedbeaf019fe>:157] ***** Predict results *****\n",
            "I0711 07:24:57.433936 139746709817216 estimator.py:612] Could not find trained model in model_dir: /content/result, running initialization to predict.\n",
            "I0711 07:24:57.496237 139746709817216 estimator.py:1145] Calling model_fn.\n",
            "I0711 07:24:57.497885 139746709817216 tpu_estimator.py:2965] Running infer on CPU\n",
            "I0711 07:24:57.499783 139746709817216 run_classifier.py:504] *** Features ***\n",
            "I0711 07:24:57.502393 139746709817216 run_classifier.py:506]   name = input_ids, shape = (?, 512)\n",
            "I0711 07:24:57.505774 139746709817216 run_classifier.py:506]   name = input_mask, shape = (?, 512)\n",
            "I0711 07:24:57.507653 139746709817216 run_classifier.py:506]   name = is_real_example, shape = (?,)\n",
            "I0711 07:24:57.509413 139746709817216 run_classifier.py:506]   name = label_ids, shape = (?, 2)\n",
            "I0711 07:24:57.511173 139746709817216 run_classifier.py:506]   name = segment_ids, shape = (?, 512)\n",
            "I0711 07:25:05.264714 139746709817216 run_classifier.py:539] **** Trainable Variables ****\n",
            "I0711 07:25:05.266128 139746709817216 run_classifier.py:545]   name = bert/embeddings/word_embeddings:0, shape = (28996, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.272164 139746709817216 run_classifier.py:545]   name = bert/embeddings/token_type_embeddings:0, shape = (2, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.273955 139746709817216 run_classifier.py:545]   name = bert/embeddings/position_embeddings:0, shape = (512, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.277369 139746709817216 run_classifier.py:545]   name = bert/embeddings/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.280821 139746709817216 run_classifier.py:545]   name = bert/embeddings/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.282500 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.288500 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.289963 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.293216 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.295526 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.297229 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.300852 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.303396 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.306503 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.309315 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.311033 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.314167 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.317102 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.319946 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.322837 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.325794 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.328989 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.331864 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.335583 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.338586 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.341605 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.344548 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.347469 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.350536 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.353408 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.356416 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.359168 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.362370 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.363610 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.365778 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.367818 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.372473 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.375252 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.378338 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.381192 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.384049 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.388890 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.391848 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.394859 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.397725 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.400544 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.404891 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.407701 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.410558 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.413486 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.416348 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.419251 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.422868 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.425834 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.428917 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.432053 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.435478 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.438683 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.441706 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.444576 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.447658 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.450504 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.453341 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.456707 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.459739 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.462627 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.465564 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.468807 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.471402 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.474852 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.476628 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.477736 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.481478 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.482677 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.483720 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.486718 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.493091 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.496382 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.499574 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.501014 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.503604 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.505403 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.506891 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.508940 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.510772 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.512584 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.514374 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.515898 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.519237 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.520891 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.522474 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.524596 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.526648 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.528894 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.530817 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.532466 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.534373 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.536241 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.538914 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.540780 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.542176 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.544476 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.546363 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.552434 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.555134 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.556711 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.558993 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.560878 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.563460 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.565788 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.567584 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.570633 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.573354 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.578972 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.581101 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.583206 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.586801 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.589291 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.591944 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.593796 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.596663 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.598865 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.601032 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.603632 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.606326 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.608576 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.610747 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.613185 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.616009 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.618391 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.621355 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.623311 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.625965 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.628565 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.630389 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.633209 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.635102 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.637610 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.639750 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.642081 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.644428 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.646553 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.648585 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.650757 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.653034 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.655670 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.658092 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.660337 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.662709 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.664695 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.666946 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.668689 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.671320 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.673748 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.675579 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.679107 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.681133 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.684334 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.686311 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.689167 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.691579 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.694293 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.696155 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.698683 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.700834 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.702495 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.705233 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.707062 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.709148 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.711313 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.713361 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.716173 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.717972 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.720679 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.723074 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.724556 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.727169 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.729898 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.731692 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.734035 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.736879 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.738760 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.740433 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.743030 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.745532 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.747761 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.750330 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.752490 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.759896 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.764298 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.768328 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.770329 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.772547 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.775388 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.777404 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.780076 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.782295 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.784797 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.787225 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.789555 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.791671 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.793366 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.800956 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.803073 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.805567 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.807701 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.809735 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.812201 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.814444 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.816937 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.818979 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.820717 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.823956 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_12/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.825972 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.828382 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.830785 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.833372 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.835217 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.837625 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.839648 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.841855 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.844456 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.845782 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.847079 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.848583 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.849953 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.851804 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.852988 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.854279 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_13/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.855943 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.857147 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.859534 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.861402 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.864337 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.866664 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.868700 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.871089 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.872481 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.874672 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.875664 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.876912 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.879823 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.882169 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.884587 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.887212 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_14/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.889561 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.891499 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.894608 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.896826 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.899046 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.901206 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.903633 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.906560 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.908092 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.910850 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.913464 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.915575 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.918478 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.919579 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.921657 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.924014 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_15/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.926664 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.929250 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.931607 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.933370 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.935945 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.938030 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.940723 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.942923 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.945168 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.947803 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.950001 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.952662 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.954513 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.957489 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.960544 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.962704 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_16/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.964796 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.967413 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.969786 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.972055 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.974301 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.976782 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.979610 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.981988 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.983865 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.986996 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.989847 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.992305 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.995460 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:05.997235 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.000082 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.002228 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_17/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.004923 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.006782 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.009854 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.012558 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.014854 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.017817 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.019556 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.022320 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.024441 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.027076 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.029131 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.031940 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.034662 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.037175 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.039582 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.041404 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_18/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.044058 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.046227 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.048880 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.051963 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.053549 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.056418 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.058717 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.061656 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.063630 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.066955 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.069025 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.071764 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.073846 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.076184 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.078919 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.081212 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_19/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.084156 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.086138 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.088767 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.091538 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.093820 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.095709 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.098011 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.100682 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.102871 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.105170 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.107852 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.110249 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.113244 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.114805 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.117582 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.120611 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_20/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.123137 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.125611 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.128249 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.130740 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.133346 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.135183 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.138088 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.140419 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.142219 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.145231 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.147138 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.149234 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.151420 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.153770 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.156347 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.158077 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_21/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.160204 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.161547 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.162856 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.164165 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.165508 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.167292 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.169410 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.170717 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.172427 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.174089 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.175108 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.176968 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.178719 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.181679 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.183514 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.185295 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_22/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.187841 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.189958 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.191662 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.194564 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.197623 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.199443 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.201183 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.203332 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.205164 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.208199 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.210563 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.213680 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.215236 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.217412 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.219609 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.221755 139746709817216 run_classifier.py:545]   name = bert/encoder/layer_23/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.223799 139746709817216 run_classifier.py:545]   name = bert/pooler/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.226372 139746709817216 run_classifier.py:545]   name = bert/pooler/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.228318 139746709817216 run_classifier.py:545]   name = output_weights:0, shape = (2, 1024), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.231087 139746709817216 run_classifier.py:545]   name = output_bias:0, shape = (2,), *INIT_FROM_CKPT*\n",
            "I0711 07:25:06.234464 139746709817216 estimator.py:1147] Done calling model_fn.\n",
            "I0711 07:25:07.948240 139746709817216 monitored_session.py:240] Graph was finalized.\n",
            "I0711 07:25:10.336503 139746709817216 session_manager.py:500] Running local_init_op.\n",
            "I0711 07:25:10.490784 139746709817216 session_manager.py:502] Done running local_init_op.\n",
            "I0711 07:25:12.526434 139746709817216 error_handling.py:96] prediction_loop marked as finished\n",
            "I0711 07:25:12.528554 139746709817216 error_handling.py:96] prediction_loop marked as finished\n",
            "I0711 07:25:12.534953 139746709817216 <ipython-input-6-fedbeaf019fe>:169] ***********REVIEW**************\n",
            "I0711 07:25:12.538537 139746709817216 <ipython-input-6-fedbeaf019fe>:170] The paper was rather bad that I don't want to see it again. The idea was trivial and the evaluations are not convincing to me at all. We should reject this paper or I won't review for this venue in the future,\n",
            "I0711 07:25:12.543490 139746709817216 <ipython-input-6-fedbeaf019fe>:171] ***********SCORE***************\n",
            "I0711 07:25:12.545716 139746709817216 <ipython-input-6-fedbeaf019fe>:172] paper\trecommendation\tconfidence\n",
            "I0711 07:25:12.548124 139746709817216 <ipython-input-6-fedbeaf019fe>:173] emnlp2019\t2.0113983\t3.8701797\n",
            "\n",
            "I0711 07:25:12.550667 139746709817216 <ipython-input-6-fedbeaf019fe>:174] ********************************\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGVxEnh0BMlf",
        "colab_type": "text"
      },
      "source": [
        "## Step 5: Check your score at the end of log\n",
        "\n",
        "For example:\n",
        "```\n",
        "**************REVIEW***********\n",
        "this is a very good paper, outstanding paper, brilliant paper. I have never seen such a good paper before. It was well-written and the models are novel. The evaluations are sound and the results achieve state-of-the-art performance. It should be definitely accepted or I will be angry.\n",
        "**************SCORE***********\n",
        "paper\trecommendation\tconfidence\n",
        "emnlp2019\t3.4766932\t3.4420846\n",
        "********************************\n",
        "```\n",
        "\n",
        "\n",
        "```\n",
        "**************REVIEW***********\n",
        "The paper was rather bad that I don't want to see it again. The idea was trivial and the evaluations are not convincing to me at all. We should reject this paper or I won't review for this venue in the future,\n",
        "**************SCORE***********\n",
        "paper   recommendation  confidence\n",
        "emnlp2019\t2.011398\t3.8701794\n",
        "********************************\n",
        "```\n",
        "​"
      ]
    }
  ]
}